feat(experimental): port SD3 image ReFL and remove the legacy core ReFL path - #281

Summary

Ends the two-ReFL transition: ports SD3 image ReFL onto the experimental.refl contract and deletes the legacy core path. Stacked on #210 — the diff collapses to the last commit once #210 merges.

• Port (experimental/refl/models/sd3.py + examples/sd3_pickscore_refl.yaml): the legacy DRaFT-K flow re-expressed through the package contract — diffuse_with_grad (mid/final grad window, per-sample [B] KL), decode_with_grad reusing the mainline grad decode (pixels in [0,1] for the core image scorers). Hyperparameters preserved from refl_sd3.yaml; the reward pairs the core PickScore scorer — no package-local reward code. pipeline_target is now the only thing a config swaps between WAN video ReFL and SD3 image ReFL, demonstrating the boundary is not WAN-shaped.
• Deliberate non-support: CFG under BPTT (guidance_scale must be 1.0, as the legacy path always ran) — fails loudly.
• Removed: unirl/train_refl.py, unirl/trainer/refl.py, unirl/train/refl/ (ReFLPolicy), unirl/models/draft.py (its only consumer), examples/diffusion/refl_sd3.yaml; trainer README updated. eval_suites stays (used by DiffusionTrainer/PETrainer). Rationale: ReFL bypasses the core StageAlgorithm abstraction (no replay, no advantages) and per the tier design lives in experimental/; net core reduction ≈ −600 lines.

Test Plan

Static (CPU, at this head): compose-check ×3 (sd3_pickscore_refl, both WAN configs) rc=0; scripts/check_recipe_targets.py — 2311 paths resolve (no dangling refl dotpaths); repo-wide grep confirms zero references to the removed modules; pre-commit run --all-files green.

Pending GPU before merge:

• sd3_pickscore_refl smoke — DONE, see the smoke comment (10/10, clean exit, PickScore-range rewards, grads flow, ~1.3s/step) (8 GPUs, ~10 rollouts): reward rises as the legacy path did; validates the port end to end.

Compatibility / Risk

• Launch surface change: python -m unirl.train_refl is gone; the replacement is python -m experimental.refl.run --config-name=sd3_pickscore_refl. No deprecation shim by design (experimental-tier policy: fail loudly).
• The removed modules had no importers outside the deleted path (verified by grep + the boundary rules).

AI-assisted; reviewed and directed by the maintainer.

Reward curve — effect-level validation

200-rollout sd3_pickscore_refl on the pinned stack (torch 2.11.0+cu128, transformers 5.6.2, peft 0.20, fp32 LoRA master), 8xH20, head 9edcab00:

The port reproduces the legacy-path curve point-for-point, so the deletion of the core ReFL path below is a like-for-like replacement. Checkpoints at rollout 50/100/150/200 retained (outputs/sd3_gate/ckpts). The dashed lines in the plot are the #120 baseline's first-10 / last-10 means.

Capability deltas vs the deleted legacy path (deliberate cuts)

A reviewer comparing this package against the deleted unirl/trainer/refl.py + unirl/train/refl/policy.py textually will find low similarity — expected: this is a like-for-like behavioral replacement on the second-generation Sample-native implementation (equivalence shown by the curve above), not a file move. The capability inventory against the legacy path:

• Supersets: DRaFT-K → arbitrary mid_timestep/final_timestep grad window; reward_loss_scale → reward_weight/baseline/scale; KL → per-sample (DP-correct).
• Deliberately cut (tier minimalism — features return with the recipe that needs them; listed in the package README): periodic eval + eval_rewards multi-suite (feat(trainer): add periodic eval to PE / ReFL / UnifiedModel #202), reward_fraction cross-slab reward placement, param_checksum probe, activation_checkpoint_vae knob (unconditionally on).